1. Field of the Invention
Methods and apparatuses consistent with the present invention relate to illumination compensation, and more particularly, to illumination compensation for video encoding and decoding.
2. Description of the Related Art
When multi-view coding (MVC) for three-dimensional (3D) display applications is performed between adjacent views during prediction between the adjacent views, changes in illumination may occur due to an incompletely calibrated camera, different perspective projection directions, different reflection effects, and the like, causing degradation in coding efficiency. Singe-view coding may also lead to degradation in coding efficiency during scene switching due to an illumination change.
To solve this problem, H.264 adopts weighted prediction. A weighted prediction scheme is applied to motion compensation at a slice level and illumination is compensated for according to an appropriate weighted factor W and an additional offset 0. Illumination change-adaptive motion estimation/motion compensation (ICA ME/MC) is an improvement on the weighted prediction scheme.
ICA ME/MC is performed in units of a 16×16 block for Y components and a differential value of illumination change (DVIC) is obtained for each macroblock.
ICA ME/MC has two modes: one is an IC-inter 16×16 mode that uses ICA ME/MC and is used for predictive (P) or bidirectional predictive (B) slices and the other is an IC-direct 16×16 mode that does not use ICA ME and is used only for B slices. In order to compensate for a local illumination change, a 1-bit flag, i.e., mb_ic_flag, is required for each of the IC-inter 16×16 mode and the IC-direct 16×16 mode.
Since DVIC for the current block has a high correlation with DVIC for neighboring blocks that are adjacent to the current block, DVIC for the current block is performed by encoding differences between DVIC for the current block and DVIC for the neighboring blocks.
Hereinafter, ICA ME/MC performed in units of a macroblock in the IC-inter 16×16 mode will be described.
For ICA ME/MC, a new sum of absolute differences (SAD) has to be defined. When a pixel at (i, j) in the current frame is referred to as f(i, j) and a pixel at (i, j) in the reference frame is referred to as r(i, j), an SAD for S×T blocks is calculated as follows, in which S×T may be 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, or 4×4.
                                          SAD            ⁢                                                  ⁢                          (                              x                ,                y                            )                                =                                    ∑                              i                =                m                                            m                +                S                -                1                                      ⁢                                          ∑                                  j                  =                  n                                                  n                  +                  T                  -                  1                                            ⁢                                                                                    f                    ⁡                                          (                                              i                        ,                        j                                            )                                                        -                                      r                    ⁡                                          (                                                                        i                          +                          x                                                ,                                                  j                          +                          y                                                                    )                                                                                                                          ,                            (        1        )            where (x, y) indicates a candidate motion vector and (i, j) indicates the position of the current block.
To compensate for an illumination change, a new SAD is required as follows:
                                          M            cur                    =                                    1                              S                ×                T                                      ⁢                                          ∑                                  i                  =                  m                                                  m                  +                  S                  -                  1                                            ⁢                                                ∑                                      j                    =                    n                                                        n                    +                    T                    -                    1                                                  ⁢                                  f                  ⁡                                      (                                          i                      ,                      j                                        )                                                                                      ⁢                                  ⁢                                                            M                ref                            ⁡                              (                                  p                  ,                  q                                )                                      =                                          1                                  S                  ×                  T                                            ⁢                                                ∑                                      i                    =                    p                                                        p                    +                    S                    -                    1                                                  ⁢                                                      ∑                                          j                      =                      q                                                              q                      +                      T                      -                      1                                                        ⁢                                      r                    ⁡                                          (                                              i                        ,                        j                                            )                                                                                                    ,                                    (        2        )            where Mcur indicates an average of pixels values in the current block, Mref indicates an average of pixels values in the reference block, and (p, q) indicates the position of the reference block. A new SAD, i.e., NewSAD (x, y), is given as follows:
                              NewSAD          ⁡                      (                          x              ,              y                        )                          =                              ∑                          i              =              m                                      m              +              S              -              1                                ⁢                                    ∑                              j                =                n                                            n                +                T                -                1                                      ⁢                                                                          {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          M                      cur                                                        }                                -                                  {                                                            r                      ⁡                                              (                                                                              i                            +                            x                                                    ,                                                      j                            +                            y                                                                          )                                                              -                                                                  M                        ref                                            ⁡                                              (                                                                              m                            +                            x                                                    ,                                                      n                            +                            y                                                                          )                                                                              }                                                                                                      (        3        )            
A block, e.g., a 16×16 block, that minimizes NewSAD (x, y) based on Equation 3 is searched for and a motion vector corresponding to the found block is also searched for.
Once a motion vector MV(x′, y′) that minimizes NewSAD (x, y) is found, an illumination compensation residual signal NewR(i, j) is determined as follows:
                                                                        NewR                ⁡                                  (                                      i                    ,                    j                                    )                                            =                            ⁢                                                {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          M                      cur                                                        }                                -                                  {                                                                                                                                          r                            ⁡                                                          (                                                                                                i                                  +                                                                      x                                    ′                                                                                                  ,                                                                  j                                  +                                                                      y                                    ′                                                                                                                              )                                                                                -                                                                                                                                                                                          M                            ref                                                    ⁡                                                      (                                                                                          m                                +                                                                  x                                  ′                                                                                            ,                                                              n                                +                                                                  y                                  ′                                                                                                                      )                                                                                                                                }                                                                                                        =                            ⁢                                                {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                -                                                                                                      ⁢                              {                                                      M                    cur                                    -                                                            M                      ref                                        ⁡                                          (                                                                        m                          +                                                      x                            ′                                                                          ,                                                  n                          +                                                      y                            ′                                                                                              )                                                                      }                                                                                        =                            ⁢                                                {                                                            f                      ⁡                                              (                                                  i                          ,                          j                                                )                                                              -                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                -                DVIC                                                                        (        4        )            
In order to indicate whether ICA ME/MC is used, the 1-bit flag, mb_ic_flag, is stored in a syntax. A differential pulse code modulation (DPCM) value for DVIC is also included in the syntax. If mb_ic_flag is ‘0’, it indicates that ICA MC is not performed for the current block. If mb_ic_flag is ‘1’, it indicates that ICA MC is performed for the current block.
When mb_ic_flag is ‘1’, an ICA ME/MC unit of a decoder obtains a reconstructed pixel as follows:
                                                                                          f                  ′                                ⁡                                  (                                      i                    ,                    j                                    )                                            =                            ⁢                                                {                                                                                    NewR                        ″                                            ⁡                                              (                                                                              x                            ′                                                    ,                                                      y                            ′                                                    ,                          i                          ,                          j                                                )                                                              +                                          r                      ⁡                                              (                                                                              i                            +                                                          x                              ′                                                                                ,                                                      j                            +                                                          y                              ′                                                                                                      )                                                                              }                                +                                                                                                      ⁢                              {                                                      M                    cur                                    -                                                            M                      ref                                        ⁡                                          (                                                                        m                          +                                                      x                            ′                                                                          ,                                                  n                          +                                                      y                            ′                                                                                              )                                                                      }                                                                                                        =                                ⁢                                                      {                                                                                            NewR                          ″                                                ⁡                                                  (                                                                                    x                              ′                                                        ,                                                          y                              ′                                                        ,                            i                            ,                            j                                                    )                                                                    +                                              r                        ⁡                                                  (                                                                                    i                              +                                                              x                                ′                                                                                      ,                                                          j                              +                                                              y                                ′                                                                                                              )                                                                                      }                                    +                  DVIC                                            ,                                                          (        5        )            where NewR″(i, j) indicates a reconstructed illumination compensation residual signal and f′(i, j) indicates a reconstructed pixel in the current frame.
Since DVIC information has to be transmitted according to conventional ICA ME/MC, coding efficiency is degraded.